Is there a way to use different vimrc file for different directory?

I guess you could use any local vimrc plugin as long as your plugin manager supports on-the-fly loading of plugins.

The almost related Q/A here is missing a few contestants: How to make vimrc settings applicable for a directory tree only. The one on SO is a little bit more complete: https://stackoverflow.com/questions/456792/vim-apply-settings-on-files-in-directory/456889#456889

Note that we cannot use different colorthemes simultaneously in a same vim session. At best the colortheme would change every time you jump into a buffer from a different project -- it would quickly become annoying IMO.