Cloudinary AWS nodejs moving files from Cloudinary to was
My role at ClickPesa is mainly on the business side by recently there was a technical problem that needed some serious coding and the tech team was busy with other responsibilities this pushed CTO Omar to ask if there is anyone that might be interested in getting their hands dirty from the business team, Without knowing what the situation was I decided to volunteer.
The situation was we had to move files mostly images from Cloudinary to AWS S3. Even if we just moved all the files at once we would have just solved the problem at once since these files in a short time they will accumulate again, so we the long-term solution would be to create a script that will be removing files from Cloudinary to AWS S3 and that could be scheduled to remove the files occasionally.
My solution approach
Researched the Cloudinary and AWS S3 APIs
The API documentation I used is
- https://cloudinary.com/documentation
- https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide
- Started by Accessing files from Cloudinary folders to my script
- Road blocker 1: Cloudinary has pagination and there is no direct way to access specific pages
- Solution: Create a script that checks if there are other files in the folder and makes the next API call until there are no more files
Upload files from my script to the AWS S3 bucket
- Create folders programmatically
- I moved the codes to the Bitbucket repository
- Created a pipeline for the script to be executed on schedules
- Connected the pipeline to Doppler
Future Updates
- Check if a bucket exists on AWS S3 if does not create one
