I needed some metadata on my S3 objects, so I had to override the S3 file system provided by flink.

I followed this guide to the letter and now I have a custom file system which works on my local machine, when I run my application in the IDE.

Now I am trying to use it on a local kafka cluster OR on my docker deployment, and I keep getting this error Could not find a file system implementation for scheme 's3c'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.

I package my application using shadowJar, using the following configuration:

shadowJar {
  configurations = [project.configurations.flinkShadowJar]
  mainClassName = "dev.vox.collect.delivery.Application"
  mergeServiceFiles()
}

I have my service file in src/main/resources/META-INF/services/org.apache.flink.core.fs.FileSystemFactory that contains a single line with the namespace and name of my factory :dev.vox.collect.delivery.filesystem.S3CFileSystemFactory

If I unzip my shadowJar I can see in its org.apache.flink.core.fs.FileSystemFactory file it has both my factory and the others declared by Flink, which should be correct:

dev.vox.collect.delivery.filesystem.S3CFileSystemFactory
org.apache.flink.fs.s3hadoop.S3FileSystemFactory
org.apache.flink.fs.s3hadoop.S3AFileSystemFactory

When I use the S3 file system provided by flink everything works, it is just mine that does not.

I am assuming the service loader is not loading my factory, either because it does not find it or because it is not declared correctly. How can I make it work? Am I missing something?