A series of UV-curable nanocomposite coating materials were prepared by sol-gel technique from tetraethoxysilane (TEOS), methacryloxypropyltrimethoxysilane (MAPTMS) in the presence of urethane acrylate resin based on polyethylene glycol 400 (PEG400). The sol-gel precursor content in the hybrid coatings was varied from 0 to 30 wt.%. In addition, acrylated phenylphosphine oxide oligomer (APPO) is replaced with urethane acrylate resin in order to investigate its effect on the nanocomposite property. The physical and mechanical properties such as; gel content, hardness, adhesion, gloss, impact strength as well as tensile strength were examined. Results from these measurements showed that all the properties of the hybrid coatings improved effectively by gradual increase in sol-gel precursor and APPO resin content. The real time infrared technique was used to follow the degree of acrylic double bond conversion. The thermal stabilities of the UV-cured nanocomposites were investigated by thermogravimetric analysis. The results revealed that the addition of sol-gel precursor and APPO oligomer into the organic network leads to an improvement in the thermal and flame resistance properties of the hybrid materials. It was also determined that the APPO containing hybrid coating with 20 wt.% silica content gave higher char yield than the coating without APPO. It is a desirable achievement to improve simultaneously both the flame retardancy and mechanical properties of a protective coating. SEM studies indicated that inorganic particles were dispersed homogenously through the organic matrix. The hybrids were nanocomposite. It was also found that, incorporation of APPO resin might govern the silica organization and this leading to formation of nanofibrillar structure.