Due to the high flammability of the paper, its use is restricted in advanced applications. In this study, UV-cured paper coating formulations containing silica nanoparticles and phosphinoxide were prepared and coated on paper surface. Flammability, wettability, and printability properties were investigated. For this purpose, the surface of the silica nanoparticles was first functionalized with hydroxyethyl methacrylate (HEMA) -OH. The hydroxylated silica nanoparticles were then modified with isocyanatoethyl methacrylate. Bis fluoro phenylphosphine oxide was synthesized by Grignard reaction elsewhere and OH modified in basic medium. Acrylate groups were added to BHPPO with isocyanatoethyl methacrylate to make it suitable for UV-curing formulation. The chemical structures of the obtained substances were illuminated by ATR-FTIR. UV-cured paper coating formulations containing acrylated silica, acrylated phosphinoxide, and mixtures thereof were prepared and coated on the paper surface. Chemical structure, contact angle, surface energies, surface morphology, thermogravimetric analysis, and limited oxygen index of the coatings were determined. As a result, it has been concluded that coated papers have increased thermal stability in the nitrogen atmosphere. The highest thermal stability was observed in coatings containing nanosilica and it was determined that all of the coatings ignited later. Hybrid coatings made within the scope of this study have improved the printability properties as well as adding flame retardancy properties to the paper.