In this present investigation, machinability studies on novel aluminum composite with hybrid reinforcements of copper-coated 4% carbon fibers (CFs) and 3% nanoclay in AA6026 matrix fabricated by compocasting method is performed. Step drill bit and multifaceted drill bit are used by adopting central composite design (CCD) in response surface methodology (RSM). The outcomes show that, with a rise in rotational speed surface irregularities, resultant force and material removal rate (MRR) intensifies, and with the additional rise in rotational speed, all the outputs decrease considerably. High MRR, resultant cutting force, and surface roughness are obtained with multifaceted carbide drills, compared with a step drill. Desirability function is used to maximize the MRR and minimize the resultant cutting forces considering the constant surface roughness of 3 μm. The optimal values are rotational speed of 1285 rpm, feed rate of 60 mm/min with the step drill bit, producing an MRR of 0.0439 kg/sec and a resultant cutting force of 185.818 N. The second-order empirical models are developed for outputs, which are fed into the non-traditional metaheuristic Evaporation Rate-based Water Cycle Algorithm (ER-WCA) therefore the lower objective value is achieved with step drill of 51.7421. It is found that using a step drill the machinability performance of this hybrid nanocomposite is well improved than the machining with other drill bits. This composite fulfills the norms of 2000/53/CE-ELV European environmental directives.