Appsync with OIDC Authorization with firebase always throwing Unauthorized for Lambda mutation

I am trying to get firebase and Appsync working together. For appsync setup I am doing:

authorizationConfig: {
  defaultAuthorization: {
    authorizationType: AuthorizationType.OIDC,
    openIdConnectConfig: {
      oidcProvider: 'https://securetoken.google.com/<with my proper project id>',
      clientId: '<with the app id from firebase>', 
      tokenExpiryFromAuth: 3600
    },
  },
  additionalAuthorizationModes: [
    {
      authorizationType: AuthorizationType.API_KEY,
      apiKeyConfig: {
        description: 'API key used by internal services needing to access AppSync',
        expires: Expiration.after(Duration.days(364)),
        name: 'internal-service-key',
      },
    },
  ],

Executing the mutation (which is directed at a Lambda) works via API Key. However, I am always getting Unauthorized when trying it via open connect. Things I do know: The oidcProvider is right I am not 100% sure on the ClientID, i assume that is the App ID from firebase console which I have used (I have also tried everything else I could find) I successfully generate an Authorization token and use it (Appsync even reports it as expired when i do use it and its expired)

I do not see any real detail of any issue in Cloudwatch besides the Unauthorized result.

I have been banging my head against the wall for quite some time trying to determine what the issue here might be. Any direction or assistance would be greatly appreciated.

Thanks