Use background image instead of border

I have a div that needs a 5px border bottom, right and left on hover.

Even if I am using box-sizing: border-box, as to not alter the size, I am having trouble because the text inside the div moves when hovering.

I seems the border is affecting the inner padding or something.

Could I recreate this effect with a background-image done with linear gradients in css?

Thank you

Edit:

I have a div

#my-div {      
  background-color: green;
}

#my-div:hover {
  border-right: 5px solid red;
  border-bottom: 5px solid red;
  border-left: 5px solid red;        
}

Instead of having all this, I would like to do a linear-gradient that looks the same as that, so it would be

#my-div:hover {
  // something that would look like if I had done
  // border-right: 5px solid red;
  // border-bottom: 5px solid red;
  // border-left: 5px solid red;
  // background-color: green;    
  background-image: linear-gradient(...);
}

and it would look the same as the first one

#my-div {
  border: 5px solid transparent;
  border-image-slice: 1;
  border-top: none;
  margin: 1em;
  padding: .3em .7em;
  display: inline-block;
  background-color: rgba(50, 161, 205, 0.5);
  }
#my-div:hover {
  border-image-source:  linear-gradient(to right, red, orange);
  }

<div id="my-div"> text not moving on hover </div>