For x∈[n,n+1) where n is an integer, [x]=n and [x+3]=n+3.
Thus f(x)=n2−(n+3)−3=n2−n−6=(n−3)(n+2).
For f(x)<0, we need (n−3)(n+2)<0, which gives −2<n<3, i.e., n∈{−1,0,1,2}.
This corresponds to x∈[−1,3).
We can verify:
For n=−1: f=1−2−3=−4<0
For n=0,1: f=−6<0
For n=2: f=4−5−3=−4<0
For n=3: f=9−6−3=0
For n≥4, (n−3)(n+2)>0.
Therefore f(x)<0 only for x∈[−1,3).