If the internal channel is 3/4" wide, then you want 3/4" tall at the end of the internal pipe, then remove all the pipe material down to the end of the channel. The 3/4"x3/4" area of channel is now the final orifice and angled properly using the remainder of the channel to control angle of jet spray. Very similar to a log jet that uses a round orifice close to the pressure hose end and the rest of the jet channel is to control angle and spreading of jet spray and the pipe is cut out in a rectangle.
When making a square orifice the total area needs to be slightly larger than the area of a circle because it takes more pressure to run through a square orifice than a round one. It equals out that a square orifice equal to the diameter of a round orifice achieves the same flow and working volume in the jet. I have been building square orifice jets for the last 3 years so I have a very good understanding of how to do this.
A round 3/4" jet has an area of .441
A 3/4"x3/4" square jet has a total area of .5625
They will both pass the same volume at a given pressure and perform the same amount of work in the jet.
Your having problems because your fitting a round peg in a square hole. Use a square peg. ie.. forget using a drill and use a small grinder and remove the pipe material from inside the jet. You wont be disappointed with the results.