I have the site collection default set to "open in client application" and I checked the document library where this is happening. It's manually set to open in the client application.
Existing documents open in the client application, but when you click the "new" button it opens in the browser.
I haven't found another setting to change, or anything else I can do to correct this behavior. Is this just a bug in SharePoint Online?