The constantly increasing electricity and energy demand in residential buildings, as well as the need for higher absorption rates of renewable sources of energy, demand for an increased flexibility at the end-users. This need is further reinforced by the rising numbers of residential Photovoltaic (PV) and battery-storage systems. In this case, flexibility can be viewed as the excess energy that can be charged to or discharged from a battery, in response to a group objective of several such battery-storage systems (aggregation). One such group objective considered in this paper includes marketing flexibility (charging or discharging) to the Day-ahead (DA) spot market, which can provide both a) financial incentives to the owners of such systems, and b) an increase in the overall absorption rates of renewable energy. The responsible agent for marketing and offering such flexibility, herein aggregator, is directly controlling the participating batteries, in exchange to some financial compensation of the owners of these batteries. In this paper, we present an optimization framework that allows the aggregator to optimally exchange the available flexibility to the DA market. The proposed scheme is based upon a reinforcement-learning approach, according to which the aggregator learns through time an optimal policy for bidding flexibility to the DA market. By design, the proposed scheme is flexible enough to accommodate the possibility of erroneous forecasts (of weather, load or electricity price). Finally, we evaluate our approach on real-world data collected from currently installed battery-storage systems in Upper Austria.