Swift is fine. I have released a Swift app that uses the maps api pretty extensively and it works great.

As BSousa mentions, xcode can be a bit flakey sometimes (syntax parser/autocomplete crashes sometimes) however it automatically restarts so it isn't really an issue.

I agree with BSousa that this FUD seems like a cultural red flag with the company.